Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Tier-2 tests #1185

Open
wants to merge 21 commits into
base: develop
Choose a base branch
from
Open

Tier-2 tests #1185

wants to merge 21 commits into from

Conversation

guillaumevernieres
Copy link
Contributor

@guillaumevernieres guillaumevernieres commented Jun 20, 2024 •
edited
Loading

Tier-2 tests would not be part of the regular GDASApp CI and would only be executed prior to submitting a PR of dev/gdasapp into develop.

Untested.

@CoryMartin-NOAA , @RussTreadon-NOAA , could you give me a sequential list of the jjobs that you want tested? That or commit directly to my branch. What needs to be updated is here

GDASApp/test/gw-ci/CMakeLists.txt

Line 61 in 6f27a49

set(TASK_LIST) # empty list for now

and here

GDASApp/test/gw-ci/CMakeLists.txt

Line 68 in 6f27a49

set(TASK_LIST) # empty list for now

and the marine DA example is here:

GDASApp/test/gw-ci/CMakeLists.txt

Line 47 in 6f27a49

set(TASK_LIST

@RussTreadon-NOAA
Copy link
Contributor

g-w C96C48_ufs_hybatmDA CI runs the following jobs

  • 20240223 18Z half cycle
    • gdasfcst and gdasatmos_prod_f*
    • enkfgdasfcst_mem[001, 002] and enkfgdasepos*
  • 20240224 00Z full cycle - g-w CI also run gfs jobs but these can be omitted for GDASApp CI
    • gdasprep
    • gdasprepatmiodaobs
    • gdasatmanlinit
    • gdasatmanlvar
    • gdasatmanlfv3inc
    • gdasatmanlfinal
    • enkfgdasatmensanlinit
    • enkfgdasatmensanlletkf
    • enkfgdasatmensanlfv3inc
    • enkfgdasatmensanlfinal

g-w CI runs jobs after the above DA jobs finish. This is useful since it ensures the generated analysis increment files are readable by downstream jobs. However, adding these downstream jobs to GDASApp CI is too much at this point.

@CoryMartin-NOAA
Copy link
Contributor

I probably won't be able to get you a list of aero+snow jobs until early next week, so don't wait for me if you want to proceed with this PR, otherwise I can get that to you Mon or Tues

@guillaumevernieres
Copy link
Contributor Author

I updated the non-existent PR description @RussTreadon-NOAA , but the idea is to not run these as part of the regular GDASApp CI, but rather only prior to doing a PR in the global-workflow develop.

@guillaumevernieres guillaumevernieres added the hera-GW-RT-tier2 1/2 cycle forecast + DA tests label Jun 21, 2024
@guillaumevernieres guillaumevernieres added the hera-GW-RT Queue for automated testing with global-workflow on Hera label Jul 8, 2024
@emcbot emcbot added hera-GW-RT-Running Automated testing with global-workflow running on Hera and removed hera-GW-RT Queue for automated testing with global-workflow on Hera labels Jul 8, 2024
@guillaumevernieres guillaumevernieres added hera-GW-RT Queue for automated testing with global-workflow on Hera and removed hera-GW-RT-Running Automated testing with global-workflow running on Hera labels Jul 8, 2024
@emcbot emcbot added hera-GW-RT-Running Automated testing with global-workflow running on Hera and removed hera-GW-RT Queue for automated testing with global-workflow on Hera labels Jul 8, 2024
@emcbot
Copy link

emcbot commented Jul 8, 2024

Automated Global-Workflow GDASApp Testing Results:
Machine: hera

Start: Mon Jul  8 16:08:16 UTC 2024 on hfe12
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Mon Jul  8 16:55:10 UTC 2024
---------------------------------------------------
Tests:                                 *SUCCESS*
Tests: Completed at Mon Jul  8 17:18:34 UTC 2024
Tests: 100% tests passed, 0 tests failed out of 48

@emcbot emcbot added hera-GW-RT-Passed Automated testing with global-workflow successful on Hera and removed hera-GW-RT-Running Automated testing with global-workflow running on Hera labels Jul 8, 2024
@guillaumevernieres guillaumevernieres added hera-GW-RT Queue for automated testing with global-workflow on Hera and removed hera-GW-RT-Passed Automated testing with global-workflow successful on Hera labels Jul 8, 2024
@emcbot emcbot added hera-GW-RT-Running Automated testing with global-workflow running on Hera and removed hera-GW-RT Queue for automated testing with global-workflow on Hera labels Jul 8, 2024
@emcbot
Copy link

emcbot commented Jul 8, 2024

Automated Global-Workflow GDASApp Testing Results:
Machine: hera

Start: Mon Jul  8 17:37:01 UTC 2024 on hfe06
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Mon Jul  8 18:27:43 UTC 2024
---------------------------------------------------
Tests:                                  *Failed*
Tests: Failed at Mon Jul  8 18:51:07 UTC 2024
Tests: 100% tests passed, 0 tests failed out of 47
Tests: see output at /scratch1/NCEPDEV/da/role.jedipara/CI/GDASApp/workflow/PR/1185/global-workflow/sorc/gdas.cd/build/log.ctest

@emcbot emcbot removed the hera-GW-RT-Running Automated testing with global-workflow running on Hera label Jul 8, 2024
@emcbot emcbot added hera-GW-RT-Running Automated testing with global-workflow running on Hera and removed hera-GW-RT Queue for automated testing with global-workflow on Hera labels Jul 8, 2024
@guillaumevernieres guillaumevernieres added hera-GW-RT Queue for automated testing with global-workflow on Hera and removed hera-GW-RT-Running Automated testing with global-workflow running on Hera labels Jul 8, 2024
@emcbot emcbot added hera-GW-RT-Running Automated testing with global-workflow running on Hera and removed hera-GW-RT Queue for automated testing with global-workflow on Hera labels Jul 8, 2024
@emcbot
Copy link

emcbot commented Jul 9, 2024

Automated Global-Workflow GDASApp Testing Results:
Machine: hera

Start: Mon Jul  8 23:43:57 UTC 2024 on hfe03
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Tue Jul  9 00:29:42 UTC 2024
---------------------------------------------------
Tier-2 Testing: Activated
Tests:                                 *SUCCESS*
Tests: Completed at Tue Jul  9 00:58:31 UTC 2024
Tests: 100% tests passed, 0 tests failed out of 47

@emcbot emcbot added hera-GW-RT-Passed Automated testing with global-workflow successful on Hera and removed hera-GW-RT-Running Automated testing with global-workflow running on Hera labels Jul 9, 2024
@guillaumevernieres
Copy link
Contributor Author

These are the new tests added:

  Test #25: WCDA-3DVAR-C48mx500
  Test #26: WCDA-3DVAR-C48mx500_gdasfcst_202103241200
  Test #27: WCDA-3DVAR-C48mx500_enkfgdasfcst_001_202103241200
  Test #28: WCDA-3DVAR-C48mx500_enkfgdasfcst_000_202103241200
  Test #29: WCDA-3DVAR-C48mx500_gdasprepoceanobs_202103241800
  Test #30: WCDA-3DVAR-C48mx500_gdasocnanalprep_202103241800
  Test #31: WCDA-3DVAR-C48mx500_gdasocnanalbmat_202103241800
  Test #32: WCDA-3DVAR-C48mx500_gdasocnanalrun_202103241800
  Test #33: WCDA-3DVAR-C48mx500_gdasocnanalchkpt_202103241800
  Test #34: WCDA-3DVAR-C48mx500_gdasocnanalpost_202103241800
  Test #35: Aero-Snow-3DVAR-C96
  Test #36: Aero-Snow-3DVAR-C96_gdasfcst_202112201200
  Test #37: Aero-Snow-3DVAR-C96_enkfgdasfcst_001_202112201200
  Test #38: Aero-Snow-3DVAR-C96_enkfgdasfcst_000_202112201200
  Test #39: Atm-hyb-C96C48
  Test #40: Atm-hyb-C96C48_gdasfcst_202402231800
  Test #41: Atm-hyb-C96C48_enkfgdasfcst_001_202402231800
  Test #42: Atm-hyb-C96C48_enkfgdasfcst_002_202402231800
  Test #43: Atm-hyb-C96C48_gdasprep_202402240000
  Test #44: Atm-hyb-C96C48_gdasprepatmiodaobs_202402240000
  Test #45: Atm-hyb-C96C48_gdasatmanlinit_202402240000
  Test #46: Atm-hyb-C96C48_gdasatmanlvar_202402240000
  Test #47: Atm-hyb-C96C48_gdasatmanlfv3inc_202402240000
  Test #48: Atm-hyb-C96C48_gdasatmanlfinal_202402240000
  Test #49: Atm-hyb-C96C48_enkfgdasatmensanlinit_202402240000
  Test #50: Atm-hyb-C96C48_enkfgdasatmensanlletkf_202402240000
  Test #51: Atm-hyb-C96C48_enkfgdasatmensanlfv3inc_202402240000
  Test #52: Atm-hyb-C96C48_enkfgdasatmensanlfinal_202402240000
  Test #53: GFSv17-3DVAR-C384mx025
  Test #54: GFSv17-3DVAR-C384mx025_gdasfcst_202106300000
  Test #55: GFSv17-3DVAR-C384mx025_enkfgdasfcst_001_202106300000
  Test #56: GFSv17-3DVAR-C384mx025_enkfgdasfcst_000_202106300000
  Test #57: GFSv17-3DVAR-C384mx025_gdasprepoceanobs_202106300600
  Test #58: GFSv17-3DVAR-C384mx025_gdasocnanalprep_202106300600
  Test #59: GFSv17-3DVAR-C384mx025_gdasocnanalbmat_202106300600
  Test #60: GFSv17-3DVAR-C384mx025_gdasocnanalrun_202106300600
  Test #61: GFSv17-3DVAR-C384mx025_gdasocnanalchkpt_202106300600
  Test #62: GFSv17-3DVAR-C384mx025_gdasocnanalpost_202106300600
  Test #63: GFSv17-3DVAR-C384mx025_gdasocnanalvrfy_202106300600
  Test #64: GFSv17-3DVAR-C384mx025_gdasprep_202106300600
  Test #65: GFSv17-3DVAR-C384mx025_gdasanal_202106300600
  [...]

Total Tests: 77

... and as I'm seeing that list, I'm thinking we probably need a new label or a way to trigger only a subset of these.

@guillaumevernieres guillaumevernieres marked this pull request as ready for review July 9, 2024 13:33
RussTreadon-NOAA
RussTreadon-NOAA reviewed Jul 9, 2024
View reviewed changes
Copy link
Contributor

@RussTreadon-NOAA RussTreadon-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While I see Tier-2 testing was a SUCCESS , we don't see a hera-GW-RT-tier2-Passed label.

A check of labels only shows hera-GW-RT-tier2. Do we need to add Running, Failed, Passed labels for hera-GW-RT-tier2?

@guillaumevernieres
Copy link
Contributor Author

While I see Tier-2 testing was a SUCCESS , we don't see a hera-GW-RT-tier2-Passed label.

A check of labels only shows hera-GW-RT-tier2. Do we need to add Running, Failed, Passed labels for hera-GW-RT-tier2?

Sorry for the confusion @RussTreadon-NOAA . The tier-2 test actually did not run, but even if they did, only the GW-RT label would be updated. The label is just there to trigger the tier-2 test, I could rename it to tier-2 only to make it less confusing?

@RussTreadon-NOAA
Copy link
Contributor

Thanks @guillaumevernieres for the explanation. I think renaming the tier-2 test trigger label would be helpful. Would the tier-2 label trigger jobs on only Hera or would it also trigger jobs on Orion?

@guillaumevernieres
Copy link
Contributor Author

Thanks @guillaumevernieres for the explanation. I think renaming the tier-2 test trigger label would be helpful. Would the tier-2 label trigger jobs on only Hera or would it also trigger jobs on Orion?

You could trigger them on other machines, but currently the marine DA ones would fail. We've been meaning to update the yamls in the g-w ... On our todo list.

@RussTreadon-NOAA
Copy link
Contributor

Got it. The hera-GW-RT-tier2 currently works as the trigger on Hera. Given this, we can keep things simple, retain this label, and revisit (if needed) later.

@RussTreadon-NOAA RussTreadon-NOAA self-requested a review July 9, 2024 16:24
RussTreadon-NOAA
RussTreadon-NOAA previously approved these changes Jul 9, 2024
View reviewed changes
Copy link
Contributor

@RussTreadon-NOAA RussTreadon-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve.

@guillaumevernieres
Copy link
Contributor Author

Not sure what I did but the tier-2 is not triggered anymore ... Great.
I'll do some debugging while working on the g-w b-matrix pr.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@RussTreadon-NOAA RussTreadon-NOAA RussTreadon-NOAA left review comments

@CoryMartin-NOAA CoryMartin-NOAA Awaiting requested review from CoryMartin-NOAA

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
hera-GW-RT-Passed Automated testing with global-workflow successful on Hera hera-GW-RT-tier2 1/2 cycle forecast + DA tests
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@guillaumevernieres @RussTreadon-NOAA @CoryMartin-NOAA @emcbot